Planet Merb

A couple of notes of interest for the work over the past few days:

• Git is holding up surprisingly well. I’ve done daily merges from rails/rails/master into wycats/rails/master and then into wycats/rails/action_view, and have so far only run into a single conflict, which was easy to resolve. Moving Rails to git made the work we’re doing possible!
• As far as I can tell, all current cases of using render() as an API now use _render_for_parts(). This is a huge milestone. Now that the four-element tuple is the standard across the board, we can push back the path lookup to the callers and unify behind something like _render_for_template(). There’s still quite a bit of work to do in general (most notably around partials and layouts), but things are coming along quite nicely.
• I was purposely holding off keeping ActionMailer in sync as I did the refactor, because things were too much in flux for that to really be viable. As it turns out, it was relatively simple to move ActionMailer over to the new approach after everything was done. The only real irritation is the repeated use of the render() options to do mostly the same thing. I’ve moved the code that takes a partial path and converts it into the requisite paths into action_pack/common.rb so it can be used in both ActionController and ActionView. I’m not sure if this will be the final resting place but this refactor definitely pulls out some AC/AV common functionality.
• I am more convinced than ever that a common AbstractController superclass for both ActionController and ActionMailer is the way to go. This was actually the thing that originally brought me to Merb (that firm conviction), and I’m pretty convinced it’s the right approach for Rails as well. Of course, there’s still quite a bit of work to do before it’s possible to really evaluate it, so stay tuned.
• Lori has been working hard on the ORM adapter for Rails. Effectively, the adapter will work like: ActiveORM.for(@sequel_obj). If the object is compatible with the ActiveORM API (note that the names et al are still very much in flux), it will be passed through. For instance, ActiveORM.for(@ar_obj) will just return the ActiveRecord object. ActiveORM.for(@sequel_obj), however, will return a wrapper around the object that is compatible with Rails expectations. Example: ActiveORM.for(@sequel_obj).new_record?. Note that this paragraph is speculative and represents work that is still very much in flux.

I’m really looking forward to finishing the work that _render_for_parts was a placeholder for. As I go through the codebase, I’m definitely noticing some obvious perf opportunities (non-trivial perf that should show up very clearly on macro-benches) and I’m looking forward to finishing the initial refactor so I can get into some perf optimizations. I should have some initial numbers within a week or so.

Today is Monday. I usually don’t like Mondays.
Being Monday goes with waking up early, going back to work, and lots of deadlines.

However, today is a special Monday. It’s the first Monday of the year and I have a special announcement!

During the Rails/Merb merge announcement, it was mentioned that I will be joining the soon to be created “Evangelism team”.

A few people asked me what being a “Rails Evangelist” means. To reassure my parents and close friends, no, I didn’t join a new cult worshiping locomotives. However, I still think that public transportation should be improved, especially in this time of crisis (but that’s a different topic).

A technical evangelist, is usually someone who knows and uses a specific technology and thinks others should look into it. This is something I’ve been doing for Merb while being part of the core team. I initiated and helped organizing MerbCamp, re-did the wiki, started working on the merb-book, spent time looking for and listening to users, spent time with third party developers and people pushing Merb to a new level (YellowPages, Wikimedia and many others).

This interaction with the end users and the third party developers is something the entire Merb team valued a great deal and I always felt it was something the community really appreciated.

As part of the merge, it was agreed that we would push things further and have a team within the Rails team to take care of “communication”. Rails is a bigger project than Merb and communication between the dev team and the users isn’t always something easy to do.

That’s why we have formed a separate team that will help communicate and support the community better. We now even have an official page on the Rails website itself

The Rails Activists

The A-Team just got announced on the Rails blog.

Instead of being called “evangelists”, we are going to be called “activists”. I think part of the argument was that the E-Team doesn’t sound as good as the A-Team.

We started with team of 4. You might not know them yet but they all are brilliant people and I’m really glad to be working with them.

Gregg Pollack, from Rails Envy. You might remember Gregg from the Rails vs * commercials or from the Rails Envy podcasts. I’ve known Gregg for a little while and he’s someone you can rely on and always full of energy/new ideas.

Ryan Bates, mainly known for his Railscasts. I only met Ryan once in person, but I’ve always been impressed by his work (don’t tell anyone, but I secretly dreamt of having something like Railscasts but for Merb )

Mike Gunderloy. I actually did not know Mike but I have read and enjoyed his blog and have seen his work on the Rails guides. Mike is an experienced writer and developer. He joked the other day saying that he started programming before any member of the Rails team was even born. Mike is a great addition to the team and I’m looking forward to learning from his experience.

Gregg and Ryan also covered the event, you might want to check their blog posts (Gregg’s and Mike’s)

So what are we going to do?

Pretty simple. We’ve boiled it down to 2 sentences:

The mission of the Rails Activists is to empower and support the worldwide network of Ruby on Rails users. We do this by publicizing Rails, making adoption easier, and enhancing developer support.

if you prefer a few more details, here are some of the tasks we are going to work on:

• Public Relations with media of all sizes
• Ombudsman work to ensure good user-to-user support
• Community Leadership at events and conferences
• Media Organization to help create good promotional opportunities
• Website maintenance
• Documentation efforts
• Developer support

Do we need help?

Absolutely! The idea is not that we are going to do all the work. The concept of this new team is to help organize the community. We are going to build a Rails Network, a network of people involved in local Rails “evangelism”/activism, people contributing and/or translating documentation, third part developers etc…

First thing would be to join the mailing list and share your suggestions, comments, concerns, etc., with us.

Secondly, we have already set up some forums to hear your feedback.

To start off, we are asking people to let us know what they would like to see happening in the Rails3 timeframe.
We have other forums for more general feedback, but we need to work with deadlines so we can prioritize accordingly. Using the Rails3 milestone should help us focus on a short/medium term deadline. Long term and not specific suggestions are welcome in the other forums.

Finally, contact us. You can find multiple ways to do so on the activism team web page.

I’ve switched to Jekyll for managing my blog. It’s funny how things come full-circle. Once you get the power of a really good editing environment (this time for me, it’s Emacs, but it could just as easily have been vim) you stop wanting to write words anywhere else.

Jekyll doesn’t do comments, obviously, since it’s really a static site generator, so I hooked up Disqus. There’s a little Sinatra app to import comments into Disqus, but it only works via RSS, and Mephisto doesn’t export all the comments into the RSS feed, just the 10 most recent.

So I hacked it to use a local MySQL database of comments instead. It’s presented here and as a gist for your enjoyment.

# Copyright 2009 Michael Ivey, released to public domain
# Disqus guts lifted from http://github.com/squeejee/disqus-sinatra-importer
# I wanted it to run from MySQL and command line, instead of a Sinatra app

require 'rubygems'
require 'feed_tools'
require 'rest_client'
require 'json'
require 'sequel'


# Fill these in. See disqus-sinatra-importer for details on what they do
# if they're not obvious
user_api_key = ''
forum_shortname = ''
current_blog_rss = ''
db = ''
db_user = ''

disqus_url = 'http://disqus.com/api'

resource = RestClient::Resource.new disqus_url
forums = JSON.parse(resource['/get_forum_list?user_api_key='+user_api_key].get)
forum_id = forums["message"].select {|forum| forum["shortname"]==forum_shortname}[0]["id"]
forum_api_key = JSON.parse(resource['/get_forum_api_key?user_api_key='+user_api_key+'&forum_id='+forum_id].get)["message"]

db = Sequel.mysql(db, :user => db_user, :host => 'localhost')
query = "SELECT title, body, author, author_email, created_at FROM contents WHERE type = 'Comment'"

# Get all of the articles from the current blog site
articles = FeedTools::Feed.open(current_blog_rss)

db[query].each do |comment|
  comment_article_title = comment[:title]
  
  # Get the blog article for the current comment thread
  article = articles.items.select {|a| a.title.downcase == comment_article_title.downcase}[0]

  if article
    article_url = article.link  
    
    thread = JSON.parse(resource['/get_thread_by_url?forum_api_key='+forum_api_key+'&url='+article_url].get)["message"]
    
    # If a Disqus thread is not found with the current url, create a new thread and add the url.
    if thread.nil?  
      thread = JSON.parse(resource['/thread_by_identifier'].post(:forum_api_key => forum_api_key, :identifier => comment[:title], :title => comment[:title]))["message"]["thread"]
      
      # Update the Disqus thread with the current article url
      resource['/update_thread'].post(:forum_api_key => forum_api_key, :thread_id => thread["id"], :url => article_url) 
    end
    
    # Import posts here
    if resource['/create_post'].post(:forum_api_key => forum_api_key, :thread_id => thread["id"], :message => comment[:body], :author_name => comment[:author], :author_email => comment[:author_email], :created_at => comment[:created_at].strftime("%Y-%m-%dT%H:%M"))
      puts "Success: #{comment.author} on #{comment.title}"
    end
  end
end

I’ve been on “vacation” for the past few days and haven’t had a ton of time to work on continuing refactoring, but I did manage to make a bit more progress, and I figured I’d share.

In Rails 2.2, ActionView had a fair number of content negotiation responsibilities. In particular, ActionView::Base had a method called template_format that looked like this:

def template_format
  if defined? @template_format
    @template_format
  elsif controller && controller.respond_to?(:request)
    @template_format = controller.request.format.to_sym
  else
    @template_format = :html
  end
end

Effectively, ActionView had a small content-negotiation responsibility, that effectively entailed grabbing the first acceptable format and using that. Thankfully, this particular piece of code was usually supplanted by Rails’ respond_to code, which performed more proper content-negotiation, and set the template format directly: @response.template.template_format = mime_type.to_sym.

A number of other places, including the JavaScript helpers, also set the template format directly, which meant that for the most part, the template_format method was just a placeholder that was set by other parts of the system. However, there were more than a few cases where the default template_format method was getting used. Instead of having ActionView call back into the request object, I wanted to modify ActionView’s initialization so that ActionController could feed it the list of acceptable formats directly.

With some work, I was able to make the modification, weaning ActionView off of content negotiation. Interestingly, this is related to my overall refactoring, which currently uses a four-element tuple to represent templates (path, acceptable extensions, prefix, and a partial flag). Now that ActionView uses a list of formats internally, it can easily pass them into find_by_parts, which handles figuring out which template to use based on the ordered list of extensions.

Just a note: find_by_parts is almost certainly a temporary creation to facilitate refactoring. However, for the purposes of refactoring, it is a convenient way to make sure that all pieces of the puzzle are passing around compatible values. In the end, we will probably be passing around Template objects, and this intermediate refactoring step will help us get all of our ducks in a row so that we can get there.

A fun diff that sort of demonstrates where I’m going with this is:

   def _pick_partial_template(partial_path) #:nodoc:
-    if partial_path.include?('/')
-      path = File.join(File.dirname(partial_path), "_#{File.basename(partial_path)}")
-    elsif controller
-      path = "#{controller.class.controller_path}/_#{partial_path}"
-    else
-      path = "_#{partial_path}"
+    unless partial_path.include?('/')
+      prefix = (controller && controller.class.controller_path)
     end
-
-    self.view_paths.find_template(path, self.template_format)
+    view_paths.find_by_parts(partial_path, formats, prefix, true)
   end

As you can see, by normalizing all path lookup to the four-element tuple, which includes merging path prefixes and appending “_” to partials, I was able to remove almost all of the code of this method, now requiring only that the controller_path is specified as a prefix if no existing prefix is supplied as part of the path (that’s so render :partial => “foo” inside of the TestController looks for “test/_foo”).

Again, I probably won’t be able to do a ton of work over the next few days, but I’ll blog when I can. Happy holidays folks!

I spent the last day of 2008 continuing my work to simplify and streamline the render path in Rails. That work involves moving all path generation to  a single place (instead of requiring that various methods throughout the system generate paths and collect the appropriate templates), and creating a single point of entry into ActionView instead of the numerous points of entry that exist now.

While I was doing this work, I noticed something that is worth discussing in more detail. In Rails 2.x, ActionController takes calls to #render, converts them some, and then calls ActionView::Base#render. It calls into ActionView with an arbitrary hash that is more or less the same hash that was passed into ActionController, with a few changes. However, that same ActionView::Base#render method is the public-facing API for calls to render inside Rails views. As a result, any work that modifies ActionView::Base “internals” has an effect on both the public-facing API of ActionView itself as well as the quasi-private API used for inbound communication from ActionController.

“What I learned today” was that when writing code, there can be seemingly very strong (DRY-related) reasons to try to reuse code in a public interface for private purposes. Resist this temptation and create solid, simple, inter-class APIs. Trying to reuse public-facing APIs for private purposes can make future refactoring very difficult, and isn’t really worth the (in the moment) gains.

On the bright side, once I deconstructed the problem, it didn’t turn out to be very difficult to get in a refactoring groove. Effectively, the plan is to (1) create a simpler inter-class communication method; (2) slowly (very very slowly) start moving existing calls into the public method into the new method.

In case you’re still interested, some specifics:

• Define some areas of responsibility: ActionController should be responsible for collecting all request-related information and passing it on to ActionView. ActionView should be responsible for aggregating the collected information and figuring out what needs to be rendered.
• Define an API: 
• ActionController will collect (1) the template name to be generated (foo); (2) a list of acceptable formats (:html, :xml) based on the current request; (3) a prefix (the current controller name, “layouts”, or something else that makes sense); (4) whether or not the template to be rendered is a partial. Note that this format works for render :file, render :template, render :action, and even render :partial.
• ActionView will receive the inbound information through a single method (provisionally called render_for_parts, but this name will likely change) and ask its PathSet (just a list of template paths–this part is completely transparent to the user) to return a template for the parts. The PathSet will handle converting the group of components into a path, and return back a Template object representing the template at the path location.
• ActionView will then call render on the template object, and the template object will return back an appropriate string (some details in this area are currently being worked out by Josh, so stay tuned for news from him).
• Note that the ActionController=>ActionView conduit is now reduced to a single method that takes a known set of 4 arguments. This conduit works for any call into ActionController::Base#render, and shifts responsibilities to where they belong (previously, ActionView was partly responsible for determine what the template format should be; this is now moved appropriately to ActionController, and is included in the arguments passed through the conduit).
• There is still some work required to make layouts work correctly, but I’m very pleased that the approach of creating a new API first and then pointing old methods at them one at a time turned out to be a very viable way to approach the problem. What this also means is that there are currently multiple copies of some logic in the codebase, but as I am able to completely remove dependencies on the old code paths, this will no longer be the case.

Apologies if I provided too much information; a lot of this is still in flux but I figured it might be interesting to get some detail about the internals of what I’m working on. Again, you can follow along on the action_view branch of my rails fork on github. Tomorrow, I should have more details on Carl, Michael, Daniel and Lori’s work. Sneak preview: Lori is already doing a bunch of really kick-ass work on ORM agnosticism. Keep an eye on her blog for details as well.

I just wanted to remind everyone that the Merb training I’m doing in Phoenix with Matt Aimonetti is filling up nicely (with some fresh Rails faces joining the roster post-merger), and that you have but a scant few hours to get in on the deal at early bird rates ($200 discount!).

If you’re working on a Merb app, this’ll give you the opportunity to learn more about the framework and best practices for development. If you’re a Rails guy, this can be your chance to get a jump on some of the ideas and code that will be making their way into Rails3.

Looking forward to seeing everyone there!

Over the past few days, I’ve been working on refactoring ActionController and ActionView to clean up the interactions between them. You can follow along on my github fork. Some principles I’m following:

• ActionController and ActionView should work well standalone
• All request-related information should be calculated in ActionController and passed through into ActionView
• ActionView should be responsible for figuring out what path to render; ActionController should pass enough information for ActionView to figure it out.
• Based on the previous, the information passed from ActionController to ActionView will likely be: template_name, list of acceptable extensions (i.e. [:html, :xml]), prefix (usually the controller name), and partial (a boolean indicating whether the template is a partial). This is not nailed down yet, but it has so far served well.
• So far, I have unified render(:template), render(:file), and render(:action) to use this new conduit, and will be working on partials tomorrow. Partials are quite complicated so my current plan may have to change slightly when I tackle them.
• Interesting info: ActionPack has a lot of methods that call each other (somewhat circularly), so it wasn’t really possible to just replace the existing conduit in a straight-forward manner. Instead, I created the new method (currently called find_by_parts, which finds a template based on the components I discussed above) and slowly (very slowly) moved existing callers of find_by_path over to use find_by_parts. Thankfully, the Rails test suite caught the initial errors I made, a huge saving grace of the entire effort.
• A nice side-effect of moving to a single, clean API between AC and AV is that the final code should be easier to understand. Once I’m further along, I’ll post some details of how exactly the interaction works.

We also released Merb 1.0.7.1 today to fix a few issues that were outstanding as well as a 1.0.7 development mode regression. (In general, we will be using the x.x.x.x moniker for midweek releases and x.x.x for weekly releases). The issues that were fixed:

• Templates should reload again in development mode
• An issue in the bundler where gems in your local repository were getting greedily installed is fixed
• An issue that was preventing bin/thor merb:gem:install use is fixed
• The error message when gems could not be found was slightly improved

To upgrade:

• gem install merb (make sure merb-gen 1.0.7.1 is installed)
• in your app, rm -rf tasks/merb.thor
• in your app, run merb-gen thor
• you should receive a prompt asking you to override bin/common.rb. Accept the prompt.
• you’re done

Happy holidays folks!